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ABSTRACT 



This paper describes a method used to evaluate 48 medical 
residency programs at one institution. It presents the background rationale, 
logistics, and results obtained by surveying alumni, in the context of their 
current practice, for relevant data on the value of their training 
experience. Surveys were sent for 3 years to all alumni of Cleveland (Ohio) 
Clinic residencies who had graduated three years earlier. A total of 187 
(28.4 percent) of alumni responded to the survey. Consistency of results 
across the 3 years was found. Ninety-six percent of respondents replied that 
they would recommend their residency program, noting such positive qualities 
as the excellent teaching, knowledge and skills of the staff, and the variety 
of experiences and patients. Suggestions for improvement centered on learning 
and practice of primary or community/outpatient care, management or business 
skills, computer uses in medicine, and improved didactics. Qualitative 
content analysis of written comments also indicated positive evaluations of 
respondents' residency experiences, and supported suggestions for more 
business training and practice and for more responsibility given to 
residents. The methodology is seen as lending itself to direct program 
improvement on a yearly basis. (Contains 10 references.) (DB) 
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Consumer Evaluation of Educational Programs: Using Questionnaires Completed hv Alumni 



Abstract 

Curriculum development and implementation is an iterative process requiring ongoing 
feedback on the program goals to appropriately train and prepare residents to become excellent 
physicians within their chosen area. It is necessary to assess the adequacy of the programs in 
matching the needs of current and future doctors within this changing climate of medicine. 
Therefore, an evaluation of multiple residency programs requires a variety of creative and 
systematic approaches. We propose one method of accomplishing these tasks. Presented here are 
the background rationale, logistics, and results obtained from surveying alumni for relevant data on 
the value of their training experience in the context of their current practice. 



Objectives/Purposes: 

An evaluation of forty-eight approved residency programs requires a variety of creative and 
systematic approaches. Indeed this call for program evaluation is “a professional responsibility to 
appraise the quality of . . . programs, and . . . constantly seek ways of improving that quality.” 
(Cronbach and others 1980 as quoted in Worthen and Sanders, 1987) A formal definition of 
evaluation provided by Kosecoff and Fink (1982) is that it is a set of procedures to assess a 
program’s worth and to provide information about its objectives, goals, processes, effects, and costs. 
Through this process of evaluation educators are more able to minimize the chance of making 
ineffective or faulty changes and demonstrate the benefits of effective changes thereby justifying 
expenditures. (Stufflebeam & Shinkfield, 1985 as quoted in Worthen and Sanders, 1987). 

Curriculum development and implementation are iterative processes requiring ongoing 
feedback on the program goals to appropriately train and prepare residents to become competent and 
productive physicians within their chosen area. Additionally, it is necessary to assess the adequacy 
of the programs in matching the needs of current and future doctors within this changing climate of 
medicine. We propose a method of measuring the success of residency programs and eliciting 
suggestions for improvement. A key component of this method is transferring these suggestions 
and feedback to the program directors and key curriculum and program implementors. The critical 
focus is the creation of a systematic method of both collecting, distributing and using the data. 
Presented here are the background rationale, logistics, and results obtained from surveying alumni 
for relevant data on the value of their training experience in the context of their current practice. 

Perspective/Theoretical Framework: 

Our goal was to gather summative information by contacting alumni across all the 
disciplinary training programs using a written survey. Furthermore, we wanted to ensure that this 
data was given to all invested parties and could be acted upon. The survey elicited alumni ’s 
perceptions on the value of specific aspects, their opinions on the programs’ success in achieving its 
objectives, and their suggestions for improving the program. All residency programs were included 
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in the sample in order to gain a broad overview of residency training in its entirety and assess global 
goals of providing training which enables residents to become competent and productive physicians 
within their specialities. By contacting alumni of all programs, comparisons could be made 
between the various programs and between different years for tracking. This broad view was 
deemed an appropriate mode for assessing the overall goals and directions of the department of 
graduate medical education which oversees numerous residency programs. 

Traditionally training programs are evaluated by gathering information from current 
residents as they complete various components and from graduating residents in the form of an 
exit interview. We decided that a valuable addition to these data sources would be the view from 
alumni; an ideal group to offer feedback on the relevance of their training from the perspective of 
being employed in the “real world” environment, experiencing the daily activities for which they 
were trained. Indeed this technique has been previously used by the clinic and other programs 
and provides a method of obtaining data and suggestions on which to modify and improve 
programs (Vanek, 1989; Parrino, 1994; Cantor, 1993, Dale, 1991). 

A span of three years of work experience was deemed an adequate time for alumni to be 
comfortable in their new position and have experienced the demands of current practice, yet still 
be within a time period where their training experience was fresh in their minds. The data 
collection instrument was developed to assess not only the goals of the programs of three years 
prior, but also the adequacy of the training for confronting current issues in medical care and 
practice. The questionnaire items reflect the current needs of practicing physicians based on a 
literature review, including such sources as Kantor and Griner (1981) and Eggert and Parkinson 
(1994). 

Methods: 

In 1995, 1996 and 1997 surveys were sent to those alumni of all Clinic residency 
programs who graduated three years prior. Surveys were mailed one time each year from the 
Office of Alumni Affairs and included addressed, stamped return envelopes. These years of data 
served not only to accomplish the current evaluation efforts, but also provide for a strong 
baseline on which to assess future changes. 

The survey consisted of thirty-three rating scale items and three open-ended items. The 
thirty-three items were organized in three broad categories, two of which were rating aspects of 
the residency program and rating the preparation provided by the program in certain areas. Both 
item sets used a five-point likert scale from “poor” to “excellent”. The additional item category 
was “amount of exposure to the following components of your training”; rated on a four-point 
opinion scale of “don’t know”, “too little”, “appropriate amount”, and “too much”. The open- 
ended questions asked for names of clinical teachers that stood out in memory and their 
approaches or skills, comments on recommendations of their program to others, and suggestions 
for improvements of their programs. Self-completed demographic information was requested 
including primary field of practice, years of training, training program area, current practice type, 
and current state or country in which practicing. 

Survey items that held particular weight in terms of alumni data were rating the 
“relevance of my training program to my current practice” and the “adequacy of my training 
program for my current practice”. Example items chosen to assess their training in reflection of 
changing trends in medicine included judging the amount of exposure to “general (primary care) 
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emphasis in my training” and “specialty (tertiary care) emphasis in my training”. Additional 
examples of items that assess their training program in terms of current medical trends are how 
well your program prepared you in... “coordinating patient care with community services/ 
resources”, “referring patient expeditiously”, “using computerized information”, and “developing 
effective interpersonal skills with patients and other professionals”. 

Data analysis included both descriptive and inferential statistics. Descriptive statistics 
were used to assess the responses for each item and the characteristics of the return sample. 
Responses of “don’t know” were not used when calculating the mean. Comparisons between 
programs were done for those programs receiving more than ten responses through computation 
of t-tests using the three composite variables of items within a category. Differences across the 
years for the two programs with large responses were analyzed through MANOVA using the 
three composite variables. 

Results: 

A total of 187 (28.4% response rate) alumni over the three years responded to the survey 
of their residency program. Though our small response rate does limit the generalizability and 
use of our data, we feel that the consistency in results across the three years demonstrates validity 
of the results. The majority of the respondents reside in Ohio (29.9%) or the eastern United 
States (32.6%), practice in fee for service groups (42.8%) or/and academic settings (27.8%), and 
had graduated from a medicine (40.1%) or surgery (34.2%) training program. A shift in practice 
setting away from “fee for service group” was noticed in 1994 (drop from 33 to 14), reflecting 
current trends in medicine. Responses were received from graduates who reside across the world 
and practice in a variety of settings and specialities. 

Ninety-six percent of the respondents would recommend their residency program. The 
top qualities mentioned by all of the alumni were the excellent teaching, knowledge and skills of 
the staff, the variety of experiences and patients, and the good learning environment. Overall 
there are no statistically significant differences among the three years of alumni responses nor 
between medicine and surgery alumni responses. All alumni ’s overall mean scores for each of 
the rating items and their written responses and suggestions were very similar across the years. 

Analysis of those items rated by alumni over both years as “excellent” or better than 
“good” and an “appropriate amount” provides information on which specific aspects of the 
residency programs worked exceptionally well. Respondents felt that their training experience 
was highly relevant and adequate to practice (4.5, sd=.68 relevance, 4.6, sd=.67 adequate). They 
felt that the variety of patients, level of supervision, their work load, their level of responsibility, 
and amount of teaching by staff were “just right”. More variance was noticeable in the exposure 
to tertiary and primary care; the ratio of tertiary to primary care was approximately equivalent 
but somewhat weighted toward specialty practice. They also felt well prepared for board exams 
(4.5, sd=.74). Additionally, alumni rated the overall quality of the program and the clinical skills 
of the teaching staff as excellent (4.6, sd=.57 quality & 4.7 sd=.57 clinical skills). As further 
commendation for the teaching staff, alumni named more than 1 80 individual staff members as 
excellent clinical teachers who stood out in their memory and provided examples and reasons for 
their choices. 

Suggestions for improvements given by respondents and identification of those items 
which did not receive the highest ratings were useful. Suggestions for improvement centered 
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around learning and practice of primary or community/outpatient care, management or business 
skills, computer uses in medicine, and improved didactics. The one item which alumni felt they 
received only fair preparation in (2.0, sd=1.21) was managing business aspects. These results 
provided confirmation to our program administrators that recent programs’ developments are 
indeed congruent with perceived needs specified by surveyed alumni. Some of these 
developments include the following activities: the creation of a primary care track which 
incorporates training in community-based medicine; the implementation of courses for residents 
to learn about cost containment, resource utilization, contracting, and risk management; ongoing 
faculty development programs; and the requirement for all senior residents to engage in research 
activity. 

Though the quantitative analyses of the rating items were consistent, support for a solid 
baseline is further demonstrated through consistency in alumni ’s written comments. Qualitative 
content analysis on the comments from three open-ended questions was done within each 
question. Six consistent categories were mentioned under support of excellent clinical teachers. 
These included use of interactive teaching techniques such as case-based approach, role- 
modeling, being spontaneous, verbalizing techniques, and teaching throughout a procedure. 
Additional categories were conveying a strong knowledge base both in basic science and clinical 
skills/medical approach, being a good discussant and providing explanations, providing a 
supportive learning environment and being committed to teaching, motivating residents, and 
teaching compassion and good patient approach. The “do you recommend your program” 
question elicited comments similar to items on the survey, such as the variety of experiences and 
patients, the teaching of clinical faculty, and the well-rounded good emphasis of their program 
which held high expectations. A concern commonly mentioned in all residency programs of the 
balance between service and training was iterated here. For suggested improvements alumni felt 
that there should be more business training and practice, more responsibility given to residents, 
and more or better teaching from more of the staff. 

Evaluation data were provided to the chairs of the Division of Education, the Department 
of Graduate Medical Education, and the Office of Alumni Affairs. These results were further 
disseminated to program directors, alumni and residents, and a general report was published in a 
faculty newsletter. It was determined that rather than presenting only the ratings on survey 
items, written comments would have the most impact and provide the most substantial 
information on which to base program improvements. Therefore, comments from alumni which 
were specific to their training program were forwarded to the directors of all training programs. 
These were shared with the chairs of education and graduate medical education but not with 
chairs within the departments. It was felt they would then be received by the stakeholders most 
directly involved in implementation decisions. Furthermore, a letter commending an individual 
staff for being mentioned by name was mailed to every staff that alumni listed as excellent 
clinical teachers. This letter was shared with the chairs of the staff s department to highlight the 
institution’s support for education and appreciation of the extra effort put forth in this cause. 

Program directors and administrators of the residency programs use multiple sources of 
data on which to base their decisions. The alumni data provide them with a method to validate 
results from other sources. Additionally, interpretation of the alumni data must involve use of 
comments from current residents and perspectives of faculty within the program. When the 
results from alumni are disseminated, a discussion of the low response rate and interpretation in 
light of program self-collected data is included. One of our residency programs felt that more 
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detailed information can be obtained from alumni and also has developed and implemented a 
survey. It is because we systematically provide both the quantitative summaries and written 
comments with notes on ensuring validity that our administrators are able to act upon these 
results and use them in their decision making about the residency program. 

Educational Importance: 

The results of this evaluation were very useful in not only assessing the value of the 
residency training programs for medical practice after training, but also for verifying the 
appropriateness of changes made in training programs to match the needs of contemporary 
medical practice. Furthermore the results provided direction for future improvements. 

The two important aspects of this study lie within the redistribution of the results or 
closing of the loop and with the collection of a baseline on which to analyze future shifts in 
program implementation. The use of alumni evaluations is not new and has been shown to 
provide a helpful source of information from a much needed perspective of the user. What we 
have hoped to demonstrate here is the deliberate process of feeding this information directly to 
the individuals who develop and oversee the program. Through an annual collection of this 
information a very steady source of data can be produced on which to judge program changes 
and policy shifts. 

This evaluation format is important as an overview of the quality of an entire residency 
program and as a method of assessing current needs. Assessments will continue annually and 
will aid in maintaining the quality of all residency programs. Additionally, as the trends in 
medicine change, the survey will be appraised and modified in order to continually collect 
relevant information concerning the current practice needs. 
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